26  Limit Theorems

Sample mean

Definition 26.1 (Sample mean) Let X_{1}, \dots , X_{n} be a sequence of i.i.d random variables with mean \mu and variance \sigma^{2}. Then the sample mean \bar{X}_{n} is defined as

\bar{X}_{n} = \frac{ 1 }{ n } \sum_{i = 1}^{n} X_{i}.

The sample mean is a random variable since it is a function of random variables. The expectation and variance of the sample mean can be calculated

\mathbb{E}_{\bar{X}_{n}} [\bar{X}_{n}] = \mathbb{E}_{X_{1}, \dots, X_{n}} \left[ \frac{ 1 }{ n } \sum_{i = 1}^{n} X_{i} \right] = \frac{ 1 }{ n } \mathbb{E}_{X_{i}} [X_{i}] = \frac{ 1 }{ n } n \mu = \mu,

\mathrm{Var} [\bar{X}_{n}] = \mathrm{Var} \left[ \frac{ 1 }{ n } \sum_{i = 1}^{n} X_{i} \right] = \frac{ 1 }{ n^{2} } \sum_{i = 1}^{n} \mathrm{Var} [X_{i}] = \frac{ 1 }{ n^{2} } n \sigma^{2} = \frac{ \sigma^{2} }{ n }.

Since \bar{X}_{n} takes a different value for each sample, it is only a guess of the true \mu, and it is closely related to the sample size n.

Law of large numbers (LLN)

There are two versions laws of large numbers, both of which state that the the sample mean of n i.i.d random variables converges to their mean \mu, that is, as n get larger, the sample mean is getting closer to \mu.

Theorem 26.1 (Weak law of large number (WLLN)) Let \bar{X}_{n} be the sample mean of n i.i.d random variables X_{1}, \dots , X_{n} with mean \mu. Then \bar{X}_{n} converges in probability to \mu

\lim_{n \to \infty} \mathbb{P} (\lvert \bar{X}_{n} - \mu \rvert > \epsilon) = 0, \quad \epsilon > 0.

NoteProof

Let X_{1}, \dots, X_{n} be i.i.d random variables with finite mean \mu and finite variance \sigma^{2}. Then for any a > 0

\mathbb{P}_{X_{1}, \dots, X_{n}} \left( \left\lvert \frac{\sum_{i = 1}^{n} x_{i}}{n} - \mu \right\rvert \geq \epsilon \right) \leq \frac{\sigma^{2}}{n \epsilon^{2}}

Applying the Chebyshev’s inequality Theorem 24.2 over multiple random variables, we get the following for any t > 0,

\begin{aligned} \mathbb{P} \left( \left\lvert \sum_{i = 1}^{n} X_{i} - \sum_{i = 1}^{n} \mu_{i} \right\rvert \geq t \right) & \leq \frac{ \sum_{i = 1}^{n} \sigma_{i}^{2} }{ t^{2} } \\ \mathbb{P} \left( \left\lvert \sum_{i = 1}^{n} X_{i} - n \mu \right\rvert \geq t \right) & \leq \frac{ n \sigma^{2} }{ t^{2} }. \\ \end{aligned}

Setting t = n \epsilon,

\begin{aligned} \mathbb{P} \left( \left\lvert \sum_{i = 1}^{n} X_{i} - n \mu \right\rvert \geq n \epsilon \right) & \leq \frac{n \sigma^{2}}{n^{2} \epsilon^{2}} \\ \mathbb{P} \left( \left\lvert \frac{ \sum_{i = 1}^{n} X_{i} }{ n } - \mu \right\rvert \geq \epsilon \right) & \leq \frac{\sigma^{2}}{n \epsilon^{2}}. \\ \mathbb{P} \left( \lvert \bar{X}_{n} - \mu \rvert \geq \epsilon \right) & \leq \frac{\sigma^{2}}{n \epsilon^{2}}. \end{aligned}

We can get WLLN by taking the limit n \to \infty

\lim_{n \to \infty} \mathbb{P} \left( \lvert \bar{X}_{n} - \mu \rvert \geq \epsilon \right) = 0.

Theorem 26.2 (Strong law of large number (SLLN)) Let \bar{X}_{n} be the sample mean of n i.i.d random variables X_{1}, \dots , X_{n} with mean \mu. Then \bar{X}_{n} converges almost surely to \mu

\mathbb{P} (\lim_{n \to \infty} \bar{X}_{n} = \mu) = 1.

NoteProof

TODO

WLLN is form of convergence in probability, while SLLN is form of almost sure convergence. Therefore, SLLN is a stronger version than the WLLN.

Central limit theorems

Theorem 26.3 (Central limit theorem (CLT)) Let \bar{X}_{n} be the sample mean of n i.i.d random variables X_{1}, \dots , X_{n} with mean \mu and variance \sigma^{2}. If n goes to infinite, then \bar{X}_{n} follows a Gaussian distribution with mean \mu and \frac{ \sigma^{2} }{ n },

\bar{X}_{n} \sim \mathcal{N} \left( \mu, \frac{ \sigma^{2} }{ n } \right).

NoteProof

TODO

Although CLT is a form of convergence in distribution, which is known to be a weaker version of convergence than convergence in probability and almost sure convergence, it doesn’t mean that CLT is a weaker version of SLLN or WLLN.

Confidence intervals

Since \bar{X}_{n} is a continuous random variable, \mathbb{P} (\bar{X}_{n} = \mu) = 0 no matter how large n is, so it is not a useful question to ask whether \bar{X}_{n} exactly equals \mu. Instead, we use confidence intervals to ask whether \mu falls within some margin of \bar{X}_{n}.

Definition 26.2 (Confidence interval) Let X_{1}, \dots , X_{n} be a sequence of random variables with mean \mu. A confidence interval for \mu with confidence level 1 - \alpha, \alpha \in (0, 1), is an interval [L, U], computed from X_{1}, \dots, X_{n}, that satisfies

\mathbb{P} (L \leq \mu \leq U) = 1 - \alpha.

Corollary 26.1 (Confidence interval for the mean) Let \bar{X}_{n} be the sample mean of n i.i.d random variables X_{1}, \dots , X_{n} with mean \mu and variance \sigma^{2}. Then a confidence interval for \mu with confidence level 1 - \alpha is [L, U] with

L = \bar{X}_{n} - z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} }, \qquad U = \bar{X}_{n} + z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} },

where z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} } is called the margin of error.

NoteProof

By Theorem 26.3, as n \to \infty,

\bar{X}_{n} \sim \mathcal{N} \left( \mu, \frac{ \sigma^{2} }{ n } \right),

and by Corollary 22.2,

Z = \frac{ \bar{X}_{n} - \mu }{ \sigma / \sqrt{n} } \sim \mathcal{N} (0, 1).

By Corollary 22.3,

\begin{aligned} 1 - \alpha & = \mathbb{P} \left( -z_{\alpha / 2} \leq Z \leq z_{\alpha / 2} \right) \\ & = \mathbb{P} \left( -z_{\alpha / 2} \leq \frac{ \bar{X}_{n} - \mu }{ \sigma / \sqrt{n} } \leq z_{\alpha / 2} \right) \\ & = \mathbb{P} \left( -z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} } \leq \bar{X}_{n} - \mu \leq z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} } \right) & [\text{multiplying by } \frac{ \sigma }{ \sqrt{n} }] \\ & = \mathbb{P} \left( \bar{X}_{n} - z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} } \leq \mu \leq \bar{X}_{n} + z_{\alpha / 2} \frac{ \sigma }{ \sqrt{n} } \right) & [\text{rearranging}] \\ & = \mathbb{P} (L \leq \mu \leq U), \end{aligned}

which proves the corollary by Definition 26.2.

Once the sample size and variance of the random variables are known, the confidence interval gives the range of values that the mean \mu is likely to fall into with a certain probability (confidence level).

• For a fixed sample size, a higher confidence level results in a wider confidence interval, which means that the estimate of \mu is less precise.

• Increasing the sample size n improves the quality of the estimate, in which case the confidence interval can be made narrower while still maintaining the same confidence level.